The transition from evolutionary stability to branching: A catastrophic evolutionary shift

Evolutionary branching-resident-mutant coexistence under disruptive selection-is one of the main contributions of Adaptive Dynamics (AD), the mathematical framework introduced by S.A.H. Geritz, J.A.J. Metz, and coauthors to model the long-term evolution of coevolving multi-species communities. It has been shown to be the basic mechanism for sympatric and parapatric speciation, despite the essential asexual nature of AD. After 20 years from its introduction, we unfold the transition from evolutionary stability (ESS) to branching, along with gradual change in environmental, control, or exploitation parameters. The transition is a catastrophic evolutionary shift, the branching dynamics driving the system to a nonlocal evolutionary attractor that is viable before the transition, but unreachable from the ESS. Weak evolutionary stability hence qualifies as an early-warning signal for branching and a testable measure of the community's resilience against biodiversity. We clarify a controversial theoretical question about the smoothness of the mutant invasion fitness at incipient branching. While a supposed nonsmoothness at third order long prevented the analysis of the ESS-branching transition, we argue that smoothness is generally expected and derive a local canonical model in terms of the geometry of the invasion fitness before branching. Any generic AD model undergoing the transition qualitatively behaves like our canonical model.